Multi-modal biometric identification

ABSTRACT

At least two biometric measurement signals are generated by contact with a single individual. At least one feature is extracted from each signal of the at least two biometric measurement signals, the extracted features are combined to generate a combined biometric signal. The combined biometric signal is compared with a defined biometric signal associated with a known individual, responsive to the combined biometric signal matching the defined biometric signal, a signal is transmitted indicating that the single individual is the known individual. The biometric measurement signals can be collected by a biometric identification device worn or carried by the single individual. The processing may be done by the biometric identification device or a remote server.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application is a continuation of U.S. patent applicationSer. No. 14/869,088 filed on Sep. 29, 2015, the content of which ishereby incorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

This disclosure relates in general to using multiple biometric signalsto identify a unique individual.

BACKGROUND

Biometric characteristics have been used to identify a unique individualfor various purposes, including but not limited to access control. Thecharacteristics conventionally include fingerprints, DNA, eye retinas,facial recognition, etc.

SUMMARY

Conventional uses of biometric characteristics for identification aretime-consuming and/or require specialized equipment. This limits theiruse for real-time applications at any number of locations where theremay be a desire to identify a unique individual.

The teachings herein may be used with a device worn by an individualthat confirms the unique identity of the individual using multi-modalbiometric identification. In this way, biometric characteristics can beused in a wide variety of applications in a speedy manner without theuse of complicated and/or expensive specialized equipment at uselocations.

According to one aspect of the teachings herein, a method includesreceiving, at a computing device, at least two biometric measurementsignals generated by contact with a single individual, wherein the atleast two biometric measurement signals comprise an electrocardiograph(ECG) signal and a photoplethysmography (PPG) signal; combining at leastone feature extracted from each signal of the at least two biometricmeasurement signals to generate a combined biometric signal; comparingthe combined biometric signal with a defined biometric signal associatedwith a known individual; and transmitting a signal, responsive to thecombined biometric signal matching the defined biometric signal,indicating that the single individual is the known individual.

According to another aspect of the teachings herein, a method includesreceiving, at a computing device, at least two biometric measurementsignals generated by contact with a single individual, wherein the atleast two biometric measurement signals comprise an electrocardiograph(ECG) signal and a photoplethysmography (PPG) signal; generating acombined biometric signal by combining the at least two biometricmeasurement signals; comparing the combined biometric signal with adefined biometric signal associated with a known individual; andtransmitting a signal, responsive to the combined biometric signalmatching the defined biometric signal, indicating that the singleindividual is the known individual.

According to another aspect of the teachings herein, a method includesreceiving, at a computing device, at least two biometric measurementsignals generated by contact with a single individual, wherein the atleast two biometric measurement signals comprise an electrocardiograph(ECG) signal and a photoplethysmography (PPG) signal; generating acombined biometric signal by combining the at least two biometricmeasurement signals; comparing the combined biometric signal with adefined biometric signal associated with a known individual; andtransmitting a signal, responsive to the combined biometric signalmatching the defined biometric signal, indicating that the singleindividual is the known individual.

According to another aspect of the teachings herein, an apparatusincludes a non-transitory memory; and a processor configured to executeinstructions stored in the non-transitory memory to receive, at acomputing device, at least two biometric measurement signals generatedby contact with a single individual, wherein the at least two biometricmeasurement signals comprise an electrocardiograph (ECG) signal and aphotoplethysmography (PPG) signal; generate a combined biometric signalby combining the at least two biometric measurement signals; compare thecombined biometric signal with a defined biometric signal associatedwith a known individual; and transmit a signal, responsive to thecombined biometric signal matching the defined biometric signal,indicating that the single individual is the known individual.

According to another aspect of the teachings herein, an apparatusincludes a body having at least two surfaces, an electrocardiogram (ECG)sensor including a first electrode coupled to a first surface of thebody and a second electrode coupled to a second surface of the body suchthat a single lead ECG is formed by contact of a first portion of anindividual with the first electrode and a second portion of theindividual with the second electrode, a photoplethysmography (PPG)sensor on one of the at least two surfaces electrically coupled to theECG sensor so as to activate responsive to forming the single lead ECG,and at least one communication device coupled to the body and controlledby a processor to wirelessly transmit biometric measurement signals fromeach of the ECG sensor and the PPG sensor to an external server,wirelessly receive a first signal from the external server indicative ofbiometric identification data generated from the biometric measurementsignals, and wirelessly transmit a second signal to an identificationdevice responsive to a match between the biometric identification datagenerated from the biometric measurement signals and biometricidentification data of a known individual.

Another apparatus described herein includes a non-transitory memory anda processor. The processor is configured to execute instructions storedin the memory to receive at least two biometric measurement signalsgenerated by contact with a single individual, extract at least onefeature from each signal of the at least two biometric measurementsignals, combine the at least one feature extracted from each signal togenerate a combined biometric signal, compare the combined biometricsignal with a defined biometric signal associated with a knownindividual, and transmit a signal, responsive to the combined biometricsignal matching the defined biometric signal, indicating that the singleindividual is the known individual.

Details of these implementations, modifications of theseimplementations, and additional implementations are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

The description herein makes reference to the accompanying drawingswherein like reference numerals refer to like parts throughout theseveral views, and wherein:

FIG. 1 is a diagram showing an example of a system configuration for abiometric identification device and server that may be used according toimplementations of this disclosure;

FIG. 2A is an illustration showing a perspective view of one example ofa wearable biometric identification device according to animplementation of this disclosure;

FIG. 2B is an illustration showing another perspective view of theexample of the wearable biometric identification device according toFIG. 2A;

FIG. 3 is a diagram showing an example of a hardware configuration for abiometric identification device and/or a server that may be usedaccording to implementations of this disclosure;

FIG. 4 is a flow chart showing an example of a process overview ofbiometric identification processing according to implementations of thisdisclosure;

FIG. 5 is a graph of a hypothetical output of an ECG sensor over timewith a number of fiducial points illustrated;

FIG. 6 is a graph of a hypothetical output of a PPG sensor over timewith a number of fiducial points illustrated;

FIGS. 7A-7D are graphs used to describe a frequency domain, appearancebased approach to feature extraction; and

FIG. 8 is a graph used to describe a time domain, appearance basedapproach to feature extraction.

DETAILED DESCRIPTION

In order to uniquely identify an individual using a biometriccharacteristic, that characteristic is universal, easily measured,unique and permanent. That is, a universal biometric characteristic isone that each individual possesses. It is easily measured if, e.g., itis technically easy and convenient for the individual to obtain ameasurement of the characteristic. Ideally, the biometric characteristicis unique in that no two individuals have identical measurements for thecharacteristic and is permanent in that the characteristic does notchange over time.

One possible biometric identification system can rely upon anechocardiogram (ECG). However, such a system presents many technicalchallenges. One challenge is that the shape of an ECG changes with heartrate. Moreover, ECGs are often noisy due to changes in an individual'sposition during measurement. ECG morphologies also vary within anindividual and a population, making a common measurement systemchallenging to implement. For example, ECG morphology may be differentfor users with or without cardiac symptoms. As another challenge,roughly 5% of the population has a non-pronounced Lead I ECG—that is, asingle lead ECG output produces features that are difficult to extractand hence presents difficulties in identifying an individual.

For the foregoing reasons, the use of an ECG alone is not accurateenough to guarantee that an individual is correctly identified. Such usecould also potentially confuse users with similar ECG morphology.

In contrast, the identification described herein combines multipledifferent biometric signals from respective sensors. Such a multi-modelbiometric system increases accuracy and specificity over using a singlebiometric signal.

FIG. 1 is a diagram showing an example of a system configuration 100 fora biometric identification device 110 and server 112 that may be used inimplementations of this disclosure.

Biometric identification device 110 as shown is a wearable biometricidentification device, namely a device worn around an individual'swrist. However, other devices can be used. For example, device 110 couldinstead be implemented by another wearable device such as a ring ornecklace. Alternatively, device 110 could be implemented as anotherportable device that is configured to travel with an individual, but notbe worn by the individual, such as a device similar in form to a keyfob.

A configuration of biometric identification device 110 is described inmore detail with reference to FIGS. 2A, 2B and 3. Referring first toFIGS. 2A and 2B, biometric identification device 110 is a wearable wristdevice 200. Although device 200 is shown as having a single, continuousbody joined at opposite ends by a single fastener, other structureswearable on or around an individual's wrist may be used.

Device 200 includes an electrocardiogram (ECG) component comprisingfirst and second electrodes 210 and 212 configured to measure variousaspects of the individual's heart function and related biometrics. Firstelectrode 210 is coupled to an exterior surface 220 and is not in directcontact with the individual that is wearing device 200. Second electrode212 is coupled to an interior surface 222 facing skin of the individualthat is wearing device 200, in this case the individual's wrist. Secondelectrode 212 may or may not be in contact with the wrist at all timeswhen device 200 is worn. First electrode 210 and second electrode 212are configured to identify electrical heart activity by measuring theuser's pulse and transmitting the measurement data for subsequentencoding and processing. That is, upon the individual contacting firstelectrode 210, for example with a finger, the second electrode 212, ifnot already in contact with the individual's skin, contacts the skin toform a single lead ECG sensor 214, which permits device 200 to measurethe individual's heart activity as discussed in more detail hereinafter.

Additionally included with device 200 is a photoplethysmography (PPG)component or sensor 230. PPG signals from PPG sensor 230 may be used toestimate skin blood flow using infrared light as discussed in moredetail herein. Although not shown in detail for clarity, PPG sensor 230is generally supported on a printed circuit board (PCB) mounted interiorof device 200 that also includes other circuitry such as amicrocontroller, battery management, and LED circuits. The circuitrycontrols the external components of PPG sensor 230, which include alight emitter and at least one photodetector spaced apart on interiorsurface 222. The light emitter transmits red and infrared lightsoriginating from red and infrared light emitting diodes through theindividual's skin, which lights are received by the photodetector. Uponreceipt, the photodetector transmits the measurement data for subsequentencoding and processing. Unlike with ECG sensor 214, which requirescompletion of a circuit between first electrode 210 and second electrode212, PPG sensor 230 does not require an additional step beyond wearingdevice 200 to take its measurements. In a desirable implementation,however, PPG sensor 230 is operatively connected to ECG sensor 214 sothat completion of the circuit between first electrode 210 and secondelectrode 212 also sends a signal according to any known technique tomonitor and measure the individual's skin blood flow.

Although not required, device 200 may include other components notexpressly shown. For example, further sensor components generatingbiometric signals through non-invasive techniques may be included withindevice 200. It will be apparent to one skilled in the art in view ofthis disclosure that the disposition of the further sensor componentswithin or on device 200 will depend on their specific nature, e.g.,whether a component can function only by contact with the skin of theindividual, whether one or more contacts are needed, etc.

As another example, device 200 may include display components. Onedisplay component may be an LED indicator that emits light whenbiometric identification data is being collected. Another may be adisplay configured to visually represent collected biometricidentification data. In an implementation, the display may be a singleoutput screen for visually representing all collected biometricidentification data. In another implementation, the display may be aplurality of output screens wherein each output screen visuallyrepresents a unique type of collected biometric identification data. Inanother implementation, the display may be a plurality of output screenswherein any collected biometric identification data may be visuallyrepresented on any such display. The information outputted to a displaymay be updated as biometric identification data is processed.

As mentioned, the photodetector of PPG sensor 230 may transmitmeasurement data for subsequent encoding and processing. Similarly, theECG sensor may transmit its measurement data. In the example of FIGS. 2Aand 2B, this transmission may be achieved by a communication component240. Communication component 240 permits device 200 to communicate withone or more external systems or devices, for example, to transmitbiometric identification data collected by device 200. As will bediscussed in greater detail below, communication component 240 mayassist a user by transmitting biometric identification data to a serverfor review or comparison against newer collected measurements ashistorical data. In an embodiment, communication component 240 is aBluetooth transmitter; however, communication component 240 may operateover other suitable wireless communication systems, including withoutlimitation an ultrasound transmitter. Accordingly, communicationcomponent 240 may receive incoming signals verifying an individual'sidentity. Communication component 240, or a separate communicationcomponent, may transmit a wireless signal upon verification of theindividual's identity. For example, this verification may be achieved bya microchip that is attached to an antenna (the microchip and theantenna together are called an RFID transponder or an RFID tag). Theantenna allows the chip to transmit the identification information to areader remote of the individual. The reader converts the radio wavesreflected back from the RFID tag into digital information that can thenbe passed on to computers for use in registering purchases, allowingentry through a security door, etc., based on the identification of theindividual.

Referring again to FIG. 1, server 112 may be implemented by anyconfiguration of one or more computers, such as remote server computers.For example, certain of the operations described herein may be performedby a server computer in the form of multiple groups of server computersthat are at different geographic locations and may or may notcommunicate with one another, such as by way of network 120. Whilecertain operations may be shared by multiple computers, in someimplementations different computers are assigned different operations.For example, one or more servers 112 could be used to process biometricidentification data as described hereinafter, and transmit a signal tothe biometric identification device 110 and/or elsewhere confirming ordenying a match, while one or more different servers 112 may receivesignals from, for example, a remote reader when the identity of theindividual carrying or wearing the biometric identification device 110is confirmed. The remote reader itself may also be a computer or part ofa computer.

Network 150 can be one or more communications networks of any suitabletype in any combination, including wireless networks, wired networks,local area networks, wide area networks, cellular data networks and theInternet. Biometric identification device 110 and server 112 cancommunicate with each other via network 120. In the implementationsdescribed herein, one network 150 is shown. Where more than one server112 is used in an implementation, each server 112 may be connected tothe same network 150 or to different networks 150.

FIG. 3 is a diagram showing an example of a hardware configuration 300for a biometric identification device and/or a server that may be usedaccording to implementations of this disclosure. For example, one ormore servers 112 could be implemented using hardware configuration 300.

Hardware configuration 300 can include at least one processor such as acentral processing unit (CPU) 310. Alternatively, CPU 310 can be anyother type of device, or multiple devices, capable of manipulating orprocessing information now-existing or hereafter developed. Although theexamples herein can be practiced with a single processor as shown,advantages in speed and efficiency may be achieved using more than oneprocessor.

Memory 320, such as a random access memory device (RAM) or any othersuitable type of non-transitory storage device, stores code and datathat can be accessed by CPU 310 using a bus 330. The code may include anoperating system and one or more application programs manipulatingand/or outputting the data. As will be discussed in detail below, anapplication program can include software components in the form ofcomputer executable program instructions that cause CPU 310 to performsome or all of the operations and methods described herein. Whenhardware configuration 300 is used to implement server 112, for example,an application program stored by memory 320 may implement some or all ofa process according to FIG. 4 as described in more detail below.

Hardware configuration 300 may optionally include a storage device 340in the form of any suitable non-transitory computer readable medium,such as a hard disc drive, a memory device, a flash drive or an opticaldrive. Storage device 340, when present, provides additional memory whenhigh processing requirements exist.

Hardware configuration 300 includes one or more input devices 350, suchas a keyboard, a mouse, a microphone or a gesture-sensitive inputdevice. Through an input device 350, data may be input from a user. Forexample, a gesture-sensitive input device may receive different gesturesto switch between different display modes (e.g., heart rate, time, ECG).Any other type of input device 350, including an input device notrequiring user intervention, is possible. For example, input device 350may be a communication device such as a wireless receiver operatingaccording to any wireless protocol for receiving input signals frombiometric identification device 110 when hardware configuration 300 isused to implement server 112. Input device 350 can output signals ordata indicative of the inputs to CPU 310, e.g., along bus 330.

Hardware configuration 300 also includes one or more output devices 360.Output device 360 may be a display or a speaker. If output device is adisplay, for example, it may be a liquid crystal display (LCD), acathode-ray tube (CRT), or any other output device capable of providingvisible output to an individual. In some cases, an output device 360 mayalso function as an input device 350—a touch screen display configuredto receive touch-based input, for example. Output device 360 mayalternatively or additionally be formed of a communication device fortransmitting signals. For example, output device 360 may include awireless transmitter using a protocol compatible with a wirelessreceiver of biometric identification device 110 to transmit signals fromserver 112 to biometric identification device 110.

Although FIG. 3 depicts one hardware configuration 300 that canimplement server 112, other configurations can be utilized. Theoperations of CPU 310 can be distributed across multiple machines ordevices (each machine or device having one or more of processors) thatcan be coupled directly or across a local area or other network.Memories 320 can be distributed across multiple machines or devices suchas network-based memory or memory in multiple machines performingoperations that may be described herein as being performed using asingle computer or computing device for ease of explanation. Although asingle bus 330 is depicted, multiple buses can be utilized. Further,storage device 340 can be a component of hardware configuration 300 orcan be a shared device that is accessed via a network. The hardwareconfiguration of a computing system as depicted in an example in FIG. 3thus be implemented in a wide variety of configurations.

The hardware configuration of one biometric identification device 110 isdescribed with reference to device 200 of FIGS. 2A and 2B. A moregeneralized configuration is represented by hardware configuration 300.For example, hardware configuration 300 may implement at biometricidentification device 110 where input devices 350 are a plurality ofbiometric measuring devices including but not limited to ECG componentor sensor 214 and PPG component or sensor 230 described with referenceto FIGS. 2A and 2B. Input devices 350 may also include a communicationcomponent, such as a wireless receiver, either alone or combined with acorresponding output device 360, such as a wireless transmitter.

FIG. 4 is a flow chart showing an example of a process or method 400 forprocessing biometric data collected from a biometric identificationdevice 110. The operations described in connection with method 400 canbe performed at one or more computing devices, such as device 110 orserver 112. When an operation is performed by one or more such computingdevices, it is completed when it is performed by one such computingdevice. The operations described in connection with method 400 can beembodied as a storage device in the form of a non-transitory computerreadable storage medium including program instructions executable by oneor more processors that, when executed, cause the one or more processorsto perform the operations. For example, the operations described inconnection with method 400 could be an application program stored atmemory 320 and be executable by CPU 310.

At operation 310, biometric measurement signals from at least twobiometric sensors are received. The biometric measurement signals may becollected as biometric measurement data from the sensors. In thisexample, the biometric measurement signals include at least ECG signalsand PPG signals, so the biometric sensors include at least an ECG sensorand a PPG sensor such as ECG sensor 214 and PPG sensor 230 shown inFIGS. 2A and 2B. In operation, the sensors may be activated to beginmeasurements by a variety of techniques. For example, an individualwearing biometric identification device 200 can contact first electrode210 so as to complete a circuit with second electrode 212 to producemeasurement data from ECG sensor 214. Contact may be made by touchingfirst electrode 210 with a finger from a hand other than the handwearing biometric identification device 200. Second electrode 212 iseither in contact with the individual at all times or is pressed intocontact therewith by the finger. That completed circuit can, in turn,activate PPG sensor 230 to produce measurement data from PPG sensor 230.In this way, biometric identification device 110 receives biometricmeasurement signals.

Method 400 may be performed in whole or in part by biometricidentification device 110. In this example, however, method 400 isperformed by server 112. Accordingly, the biometric measurement signalsmay be received at server 112 from biometric identification device 110through, for example, a wireless communication component in operation410. Also in this example, only ECG and PPG signals are discussed, butother biometric measurement signals would be subject to the sameprocessing. In some implementations, measuring biometric signals endsafter a defined period of time lapses. In other implementations,measuring biometric signals ends when contact of the individual withfirst electrode 210 ends—breaking the circuit formed with secondelectrode 212. It is also possible that the period for measuring somebiometric signals is longer than the period for measuring otherbiometric signals and/or these techniques are combined. For example, ECGsensor 214 obtains measurements only while contact is maintained by theindividual with first electrode 210, and PPG sensor 230 obtainsmeasurements for a predetermined time period after activation.

At operation 420, the ECG signals and PPG signals separately undergosignal pre-processing to prepare the subsequent analysis. Pre-processingencompasses a number of manipulations to the signals to ensure dataintegrity and prepare the signals for feature extraction at operation430. The type of pre-processing varies according to the type of signal,but it generally involves denoising of the raw signals from the sensors.Pre-processing may include, for example, removing baseline wander in thesignals. This processing generally adjusts the input signals during onemeasurement cycle to a common baseline. Filtering, such as using a bandpass filter, may be applied in order to remove any undesirable datashifts that occurred while the signals were being measured and to reducethe presence of data outside of a range to be observed (e.g., outliers).While the biometric signals from different sensors are processedseparately, comparisons between them may be useful for processing eachsignal. For example, where different signals each derive from theindividual's pulse measurement, the periodicity of the measured signalsis likely to be equal or substantially equal. To the extent they arenot, this may indicate motion noise within one or the other of thesignals. Motion noise may include, for example, fluxes and other changespresent in ECG and PPG signals due to the user walking, running,exercising, or otherwise moving in a manner that may interfere with aclear biometric measurement (e.g., where the user's contact with firstelectrode 210 moves while ECG signals are being measured). Thedifferences may be used in a filtering process for motion noise.

At operation 430, feature extraction occurs on each respectivepre-processed signal. There is more than one approach for featureextraction that may be used in operation 430. In general, a fiducialpoint based approach (also called a rule based approach) detects valuesassociated with various segments of a signal, while an appearance basedapproach detects the shapes of the signal. It can be the shape in thetime domain, such as a wave fragment, or the shape in the frequencydomain, for example, the output from the combination of autocorrelationand discrete cosine transform of the signal. The fiducial point basedapproach is described with reference to FIGS. 5 and 6, while theappearance based approach is described with reference to FIGS. 7A-7D and8.

FIG. 5 is a graph of a hypothetical output of an ECG sensor over timewith a number of fiducial points illustrated. The output of FIG. 5 isassumed to be a pre-processed signal resulting from operation 420 and isidealized as seen by the flat baseline. The output curve generallyrepresents voltage over time, but any appropriate measurement unit maybe used. A typical ECG output is a repeating cycle formed of a P wave(representing atrial depolarization), a QRS complex (representingventricular depolarization) and a T wave (representing ventricularrepolarization). A PR segment exists from the end of the P wave to thebeginning of the QRS complex, and an ST segment exists from the end ofthe QRS complex to the beginning of the T wave. Other electricalentities may be represented in an ECG output.

Each of these electrical entities within an ECG curve is associated withone or more amplitudes (used interchangeably with magnitude hereinunless otherwise noted) and one or more time intervals or durations. Forexample, in the QRS complex, Q and S are valleys and R is a peak, eachassociated with a different amplitude. The amplitude of any point withincan be either an absolute amplitude (measured relative to the baseline)or a relative amplitude (measured as compared to another amplitude).Using absolute measurements, for example, FIG. 5 shows that valley Q hasa Q wave magnitude, peak R has an R wave magnitude, and valley S has anS wave magnitude. The magnitude of the T wave and the magnitude of the Pwave are also shown in FIG. 5. An interval or duration may be measuredfrom any point in the repeating cycle to any other point. For example,the individual may be represented by a PR interval from the start of theP wave to the start of the QRS complex and a QT interval from the startof the QRS complex to the end of the T wave.

Using the fiducial based approach, feature extraction in FIG. 5 involvesdetecting or calculating at least some of the durations/intervals and atleast some of the amplitudes/magnitudes within the ECG curve orwaveform. Feature extraction may be achieved using calculationsincluding one or more of a spectral based feature, wavelet, discretecosine transformation (DCT), power density, Ensemble Empirical ModeDecomposition (EEMD). The features could be the amplitude and durationvalues themselves, combinations of the amplitude and/or duration valuesor values derived using the amplitude and/or duration values through,for example, Autocorrelation Coefficient (AC) or a Periodicity Transform(PT). One feature may be, for example, heart rate variability (HRV),which is the variation of beat-to-beat intervals (i.e., the time from Rto R per cycle). The features may be reduced or encoded using principalcomponent analysis (PCA), latent discriminant analysis (LDA) and/orindependent component analysis (ICA).

FIG. 6 is a graph of a hypothetical output of a PPG sensor over timewith a number of fiducial points illustrated. It is based on FIG. 7 inRe

it Kaysao{hacek over (g)}lu et al., “A novel feature ranking algorithmfor biometric recognition with PPG signals,” Computers in Biology andMedicine 49 (2014). The output of FIG. 6 is assumed to be apre-processed PPG signal resulting from operation 420. A typical PPGoutput is a repeating cycle that can have various forms other than thatshown by example in FIG. 6, which is idealized for illustrationpurposes. Like FIG. 5, the x-axis represents time (e.g., in seconds).Here, the y-axis is per-unit amplitude.

The PPG signal of FIG. 6 has a number of fiducial points useful foridentification that can be extracted at operation 430. In a cycle, thehighest point reached after a hole 600 (a baseline value) is a systolicpeak 602 at amplitude H₁. The dicrotic notch 604 has an amplitude H₂,and the diastolic peak 606 has an amplitude H₃. The diastolic peak 606is ideally located at half of the amplitude H₁/2, but it may not be inpractice. In any event the width W of the cycle is measured at amplitudeH₁/2. Time periods or durations also provide desirable data for featureextraction. The time duration t_(pp) is measured between adjacentsystolic peaks 602, while the time duration t_(hh) is measured betweenadjacent holes 600. Some other time durations shown include timeduration t₁ between a hole 600 and a systolic peak 602, time duration t₂between the systolic peak 602 and a dicrotic notch 604, and timeduration t₃ between the dicrotic notch 604 and a diastolic peak 606. Thearea under each portion of the cycle may be a feature extracted from thepoints of FIG. 6. In FIG. 6, area A₁ represents the area under the curvedefined by a systolic peak 602 between a hole 600 and a dicrotic notch604, while area A₂ represents the area under the curve defined by thediastolic peak 606 between a dicrotic notch 604 and a hole 600.Accordingly, feature extraction in FIG. 6 using the fiducial basedapproach involves detecting or calculating at least some of thedurations t₁, t₂, t₃, t_(pp), and t_(hh) and at least some of theamplitudes H₁, H₂, and H₃.

FIGS. 7A-7D are graphs used to describe a frequency domain, appearancebased approach to feature extraction. The processing of FIGS. 7A-7D isdescribed with respect to an ECG signal only, but similar processingwould apply to any cyclical biometric input signal, including a PPGsignal. Moreover, the processing of FIGS. 7A-7D is one example of anappearance based approach—others that analyze the cyclical nature of thesignal without the specific identification of multiple fiducial pointswithin the signal are also possible given the teachings herein.

FIG. 7A is a graph of voltage versus time, which represents threeseconds of an ECG signal from a person. The output of FIG. 7A is assumedto be a pre-processed ECG signal resulting from operation 420. Featureextraction at operation 430 in this case is a four-step process thatbegins with windowing, where the pre-processed ECG signal or trace issegmented into non-overlapping windows so that each window has a lengthsufficient to encompass multiple cycles. For example, each window forthe signal of FIG. 7A should be longer than the average heartbeat lengthso that multiple pulses are included. Next, autocorrelation is performedso as to estimate the normalized autocorrelation of each window. Doingso to the signal of FIG. 7A extending for several more cycles results inthe graph of FIG. 7B, which is a zoomed plot of the power versus signaltime after autocorrelation. Once autocorrelation is completed, adiscrete cosine transform over L lags of the autocorrelated signal isperformed, where L is a positive integer. FIG. 7C is a DCT plot of thepower of the resulting DCT coefficients. FIG. 7D is a zoomed DCT plot ofthe power values of truncated DCT coefficients from FIG. 7C. The finalstep of the feature extraction is classification based on thesignificant DCT coefficients. For example, the extracted features couldinclude a certain number of the highest DCT coefficients, how many DCTcoefficients are associated with certain power values, a fitted curvefor the points defining peaks or valleys in the DCT plot, etc.

FIG. 8 is a graph used to describe a time domain, appearance basedapproach to feature extraction. The time domain, appearance basedapproach to feature extraction may involve, for example, extractingPQRST fragments for characterization of an individual. This extractioninvolves, first, superimposing samples of an ECG sensor output over timeby synchronizing their peaks R. Then, a uniform fragment length for thePQRST fragments is selected. The PQRST fragments are intended to captureat least the entirety of a cardiac cycle from the start of a P wave tothe end of a T wave. Because the intervals within the cycle are notconstant, the uniform fragment length may be selected by detecting thepeak R and selecting a constant number of samples from the output of theECG sensor before the peak R and a constant number of samples from theoutput of the ECG sensor after the peak R. The number of samples beforethe peak R and the number of samples after the peak R used to generatethe PQRST fragments may be different. After extraction, the PQRSTfragments may be further processed to minimize dissimilarities to theextent they remain after pre-processing at 420. For example, thefragments may be adjusted vertically due to baseline drift, somefragments may be filtered out due to distortion during measurement(e.g., they are too different from the mean due to motion, etc.), and/orthey may be corrected due to variations in heart rate according to knowntechniques. FIG. 8 illustrates several extracted PQRST fragments afterthis processing. The fragments of FIG. 8 may then be used forclassification, either with or without a reduction in the feature spaceusing PCA, wavelet transformation, etc.

As may be apparent from the foregoing descriptions, approaches tofeature extraction each have their strengths and weaknesses. While thefiducial point based approach is well-adjusted to changes in heart rate(e.g., its accuracy does not significantly decrease with heart ratechanges), it requires a good detection algorithm, it is slow, and it issensitive to noise. In contrast, the frequency domain, appearance basedapproach does not need to detect various segments of the signal, so itis faster. The time domain, appearance based approach only relies on thedetection of the peak R of the QRS complex, which is a relatively easiertask as compared to the detection of other fiducial points. Theappearance based approach is also less sensitive to noise. Thus, andreferring again to FIG. 4, feature extraction at operation 430 uses bothapproaches on each biometric measurement signal to improve accuracyherein. Thus, feature extraction at operation 430 combines the analysisfor a single biometric signal.

There are two techniques for combining the two approaches. The first isa feature level combination and the second is a decision levelcombination. In one example of the feature level combination, featuresextracted from each approach are produced as the combined output for aninput biometric signal. In one example of the decision levelcombination, thresholds or other tests may be applied to featuresgenerated by each approach to select only some of the features as thecombined output for an input biometric signal. For example, featuresfrom each approach may be selected when a feature falls outside of oneor more standard deviations of the values for the feature within a knownpopulation. In another example, features that are not likely to bedifferentiating may be filtered out.

Referring again to FIG. 4, operation 440A or both of operations 440A and440B may be performed after feature extraction at operation 430.Multi-modal feature fusion in operation 440A combines the features fromeach of the biometric measurement modes into a common set of data foruse in a subsequent biometric identification matching step. In oneimplementation, the fusion is a learning based feature fusion methodthat applies weights associated with each modality that are learnedthrough training data to generate biometric features for the individualcurrently using the biometric identification device 110 for comparisonwith biometric features of a known owner/user.

Optionally, operation 440B in process 400 may be used for obtaining thedata for offline training. More specifically, some or all of thefeatures extracted at operation 430 may be provided to operation 440B tocombine with other data for offline training. Operation 440B may beperformed by server 112 or elsewhere and may include data for extractedfeatures collected from a relatively large population of individuals astraining data. According to one implementation, the training data fromthe different modalities is ranked using a regression technique, such asRandom Forest™ regression. Then, each feature is assigned a weight as aresult of the regression. Other learning based methods of feature fusionare also possible, such as a support vector machine (SVM) method, metriclearning, etc. Each technique desirably results in a weight thatreflects the relative importance of an extracted feature as compared toall features. For example, an extracted feature that is common to alarge number of individuals would be less important (and hence be givena lower weight) than an extracted feature that is less common and wouldprovide a better feature for differentiating one individual from thepopulation as a whole.

Regardless of whether the current output of operation 430 is providedfor offline training at operation 440B, the weights learned through thetraining data at operation 440B are used at operation 440A for themulti-modal feature fusion in conjunction with the current data. In oneimplementation of operation 440A, weights from the offline training 440Bare applied to the features extracted at operation 430 to generate datafor a comparison at operation 460.

In an initial set up process, the biometric identification device 110may be associated with an individual by performing operations 410, 420,430, and optionally 440A. The resulting output of operation 430 and/oroperation 440A may be stored at the end of the initial set up processfor the comparison at operation 460. Whether or not operation 440A isperformed at the initial set up process, an optional operation 450 canapply the weights from the offline training of operation 440B duringprocess 400 to the stored set up data before the comparison of operation460. In this way, changes in the population as a whole can be used tokeep the comparison up to date and new biometric features may be takeninto account. The data may be stored locally in the biometricidentification device 110 and accessed by server 112 by transmission orelsewise, or may be stored at server 112 for retrieval when anidentifier unique to the biometric identification device 110 istransmitted (e.g., with the signals at operation 410).

Further, in some implementations, a set up process after the initial setup process may occur by which old stored data is replaced with new data.This may be done using another authentication process according to knowntechniques, such as through the use of a personal identification number(PIN), etc., that confirms the individual's identity before updating thefeature data. This allows for use of the biometric identification device110 even after a change in the individual's biometrics, e.g., through achange in health.

Regardless of what verification data is available for verification, thecomparison of operation 460 compares that data to the output ofoperation 440A associated with the newly-measured biometric signals. Thecomparison may be done on server 112, with server 112 then sending asignal at operation 470 to biometric identification device 110 upon amatch. That signal could activate the antenna of the RFID transponder ofthe biometric identification device 110 to transmit the identificationinformation to a reader remote of the individual. Alternatively, thecomparison may be done at the biometric identification device 110. Ifthere is no match at the comparison of operation 460, no signal may besent. Alternatively, a signal may be sent indicating the lack of a matchthat, in turn, activates an alarm or otherwise provides notice to theindividual and optionally to others that there is no match. For example,a signal could be sent to a vendor at which the individual is attemptingto use the device 110 that there is no match.

A match does not require all the data from the known individual to matchthe data from the new signals. Instead, various quantities within thedata can be compared to the corresponding quantities generated from thenew signals. If the differences between the pairs of quantities are allwithin a defined range, for example, a match may be signaled.

As used herein, information, signals, or data are received bytransmission or accessing the information, signals, or data in any form,such as receiving by transmission over a network, receiving by accessingfrom a storage device, or receiving by user operation of an inputdevice.

The foregoing description describes only some implementations of thedescribed techniques. Other implementations are available. For example,the particular naming of the components, capitalization of terms, theattributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the systems and methods described herein or their features mayhave different names, formats, or protocols. Further, the system may beimplemented via a combination of hardware and software, as described, orentirely in hardware elements. Also, the particular division offunctionality between the various system components described herein ismerely exemplary, and not mandatory.

The word “example” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“example” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the word“example” is intended to present concepts in a concrete fashion. As usedin this application, the term “or” is intended to mean an inclusive “or”rather than an exclusive “or”. That is, unless specified otherwise, orclear from context, “X includes A or B” is intended to mean any of thenatural inclusive permutations. That is, if X includes A; X includes B;or X includes both A and B, then “X includes A or B” is satisfied underany of the foregoing instances. In addition, the articles “a” and “an”as used in this application and the appended claims should generally beconstrued to mean “one or more” unless specified otherwise or clear fromcontext to be directed to a singular form. Moreover, use of the term “anembodiment” or “one embodiment” or “an implementation” or “oneimplementation” throughout is not intended to mean the same embodimentor implementation unless described as such.

The implementations of the computer devices (e.g., clients and servers)described herein can be realized in hardware, software, or anycombination thereof. The hardware can include, for example, computers,intellectual property (IP) cores, application-specific integratedcircuits (ASICs), programmable logic arrays, optical processors,programmable logic controllers, microcode, microcontrollers, servers,microprocessors, digital signal processors or any other suitablecircuit. In the claims, the term “processor” should be understood asencompassing any of the foregoing hardware, either singly or incombination. Further, portions of each of the clients and each of theservers described herein do not necessarily have to be implemented inthe same manner.

Operations that are described as being performed by a single processor,computer, or device can be distributed across a number of differentprocessors, computers or devices. Similarly, operations that aredescribed as being performed by different processors, computers, ordevices can, in some cases, be performed by a single processor, computeror device.

Although features may be described above or claimed as acting in certaincombinations, one or more features of a combination can in some cases beexcised from the combination, and the combination may be directed to asub-combination or variation of a sub-combination.

The systems described herein, such as client computers and servercomputers, can be implemented using general purpose computers/processorsmodified with a computer program that, when executed, carries out any ofthe respective methods, algorithms and/or instructions described herein.In addition or alternatively, for example, special purposecomputers/processors can be utilized which can contain specializedhardware for carrying out any of the methods, algorithms, orinstructions described herein.

Some portions of above description include disclosure presented in termsof algorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are the means used bythose skilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. These operations,while described functionally or logically, are understood to beimplemented by computer programs. Furthermore, it has also provenconvenient at times, to refer to these arrangements of operations asmodules or by functional names, without loss of generality. It should benoted that the process steps and instructions of implementations of thisdisclosure could be embodied in software, firmware or hardware, and whenembodied in software, could be downloaded to reside on and be operatedfrom different platforms used by real time network operating systems.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

At least one implementation of this disclosure relates to an apparatusfor performing the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored on a computer readable storage medium that canbe accessed by the computer.

Certain portions of the embodiments of the disclosure can take the formof a computer program product accessible from, for example, anon-transitory computer-usable or computer-readable medium. The computerprogram, when executed, can carry out any of the respective techniques,algorithms and/or instructions described herein. A non-transitorycomputer-usable or computer-readable medium can be any device that can,for example, tangibly contain, store, communicate, or transport theprogram for use by or in connection with any processor. Thenon-transitory medium can be, for example, any type of disk includingfloppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-onlymemories (ROMs), random access memories (RAMs), EPROMs, EEPROMs,magnetic or optical cards, application specific integrated circuits(ASICs), or any type of media suitable for tangibly containing, storing,communicating, or transporting electronic instructions.

What is claimed is:
 1. A method, comprising: receiving, at a computingdevice, at least two biometric measurement signals generated by contactwith a single individual, wherein the at least two biometric measurementsignals comprise an electrocardiograph (ECG) signal and aphotoplethysmography (PPG) signal; combining at least one featureextracted from each signal of the at least two biometric measurementsignals to generate a combined biometric signal; comparing the combinedbiometric signal with a defined biometric signal associated with a knownindividual; and transmitting a signal, responsive to the combinedbiometric signal matching the defined biometric signal, indicating thatthe single individual is the known individual.
 2. The method of claim 1,wherein combining the at least one feature extracted from each signal ofthe at least two biometric measurement signals to generate a combinedbiometric signal comprises: selecting at least one feature from the atleast one feature extracted from each signal of the at least twobiometric measurement signals as the combined biometric signal.
 3. Themethod of claim 1, wherein combining the at least one feature extractedfrom each signal of the at least two biometric measurement signals togenerate a combined biometric signal comprises: applying a weight toeach feature extracted from at least one of the ECG signal and the PPGsignal, wherein the weight is learned from training data collected froma plurality of individuals.
 4. The method of claim 1, furthercomprising: receiving, at the computing device, at least one featureassociated with the known individual and extracted during a set upprocess; and generating, using the at least one feature, the definedbiometric signal using a same technique used to generate the combinedbiometric signal before comparing the combined biometric signal with thedefined biometric signal.
 5. The method of claim 4, further comprising:updating the defined biometric signal in response to receiving anindication that the single individual is the known individual by anotherauthentication process.
 6. The method of claim 1, wherein receiving theat least two biometric signals comprises receiving the at least twobiometric measurement signals generated by a biometric identificationdevice worn by the single individual.
 7. The method of claim 6, whereinthe biometric identification device and the computing device is the samedevice.
 8. The method of claim 1, further comprising: storing the atleast one feature from each signal of the at least two biometricmeasurement signals generated by contact with the known individual. 9.The method of claim 1, wherein the at least one feature extracted fromeach signal of the at least two biometric measurement signals comprisesat least one feature determined by at least one of: identifying a timeduration and an amplitude associated with at least one cycle of an inputsignal; performing autocorrelation and a discrete cosine transformationusing a plurality of cycles of the input signal; and extracting at leastone PQRST fragment.
 10. A method, comprising: receiving, at a computingdevice, at least two biometric measurement signals generated by contactwith a single individual, wherein the at least two biometric measurementsignals comprise an electrocardiograph (ECG) signal and aphotoplethysmography (PPG) signal; generating a combined biometricsignal by combining the at least two biometric measurement signals;comparing the combined biometric signal with a defined biometric signalassociated with a known individual; and transmitting a signal,responsive to the combined biometric signal matching the definedbiometric signal, indicating that the single individual is the knownindividual.
 11. The method of claim 10, wherein generating a combinedbiometric signal by combining the at least two biometric measurementsignals comprises: generating the combined biometric signal by selectingat least one feature from features extracted from each signal of the atleast two biometric measurement signals.
 12. The method of claim 10,wherein generating a combined biometric signal by combining the at leasttwo biometric measurement signals comprises: generating the combinedbiometric signal by combining a first feature extracted from the ECGsignal with a second feature extracted from the PPG signal.
 13. Themethod of claim 10, wherein generating a combined biometric signal bycombining the at least two biometric measurement signals comprises:applying a weight to each feature extracted from at least one of the atleast two biometric measurement signals, wherein the weight is learnedfrom training data collected from a plurality of individuals.
 14. Themethod of claim 10, further comprising: receiving, at the computingdevice, at least one feature associated with the known individual andextracted during a set up process; and generating, using the at leastone feature, the defined biometric signal using a same technique used togenerate the combined biometric signal before comparing the combinedbiometric signal with the defined biometric signal.
 15. The method ofclaim 14, further comprising: updating the defined biometric signal inresponse to receiving an indication that the single individual is theknown individual.
 16. The method of claim 10, wherein receiving the atleast two biometric signals comprises receiving the at least twobiometric measurement signals generated by a biometric identificationdevice worn by the single individual.
 17. The method of claim 16,wherein the biometric identification device and the computing device isthe same device.
 18. The method of claim 10, further comprising: storingthe at least one feature from each signal of the at least two biometricmeasurement signals generated by contact with the known individual. 19.The method of claim 10, wherein the at least one feature extracted fromeach signal of the at least two biometric measurement signals comprisesat least one feature determined by at least one of: identifying a timeduration and an amplitude associated with at least one cycle of an inputsignal; performing autocorrelation and a discrete cosine transformationusing a plurality of cycles of the input signal; and extracting at leastone PQRST fragment.
 20. An apparatus, comprising: a non-transitorymemory; and a processor configured to execute instructions stored in thenon-transitory memory to: receive, at a computing device, at least twobiometric measurement signals generated by contact with a singleindividual, wherein the at least two biometric measurement signalscomprise an electrocardiograph (ECG) signal and a photoplethysmography(PPG) signal; generate a combined biometric signal by combining the atleast two biometric measurement signals; compare the combined biometricsignal with a defined biometric signal associated with a knownindividual; and transmit a signal, responsive to the combined biometricsignal matching the defined biometric signal, indicating that the singleindividual is the known individual.